Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: parallelise loading of dag-pb links when exporting #249

Merged
merged 2 commits into from
Aug 17, 2022
Merged

fix: parallelise loading of dag-pb links when exporting #249

merged 2 commits into from
Aug 17, 2022

Conversation

achingbrain
Copy link
Member

A polishing of #237. Uses it-parallel to load a whole list of children of a DAG node in parallel rather than one at a time. Makes fetching large files much faster.

From the js-ipfs interop suite:

Before:

  exchange files
    go -> js
      cat file
        ✔ go -> js: 67.1 MB (25659ms)
    go -> go2
      cat file
        ✔ go -> go2: 67.1 MB (19835ms)
    js -> go
      cat file
        ✔ js -> go: 67.1 MB (17407ms)
    js -> js2
      cat file
        ✔ js -> js2: 67.1 MB (20237ms)

After:

  exchange files
    go -> js
      cat file
        ✔ go -> js: 67.1 MB (15328ms)
    go -> go2
      cat file
        ✔ go -> go2: 67.1 MB (18536ms)
    js -> go
      cat file
        ✔ js -> go: 67.1 MB (17211ms)
    js -> js2
      cat file
        ✔ js -> js2: 67.1 MB (12124ms)

@achingbrain achingbrain changed the title fix: only support dag-pb or raw fix: parallelise loading of dag-pb links when exporting Aug 17, 2022
@achingbrain achingbrain mentioned this pull request Aug 17, 2022
Closed
@achingbrain achingbrain merged commit 862d63b into master Aug 17, 2022
@achingbrain achingbrain deleted the fix/export-performance branch August 17, 2022 15:17
github-actions bot pushed a commit that referenced this pull request Aug 17, 2022
@semantic-release-bot
chore(release): 8.0.2 [skip ci]
57659fc 
## [ipfs-unixfs-exporter-v8.0.2](ipfs-unixfs-exporter-v8.0.1...ipfs-unixfs-exporter-v8.0.2) (2022-08-17)

### Bug Fixes

* parallelise loading of dag-pb links when exporting ([#249](#249)) ([862d63b](862d63b)), closes [#237](#237)
@github-actions
Copy link

🎉 This PR is included in version ipfs-unixfs-exporter-v8.0.2 🎉

The release is available on:

Your semantic-release bot 📦🚀

@alanshaw alanshaw mentioned this pull request Feb 28, 2023
Merged
achingbrain pushed a commit that referenced this pull request Mar 10, 2023
fix: parallelise loading of dag-pb links in directories when exporting (
9e01878 
#286)

This PR extends the work done in #249 to also include directories and sharded directories.
@alanshaw alanshaw mentioned this pull request Mar 13, 2023
Merged
@achingbrain achingbrain mentioned this pull request Mar 15, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
released
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@achingbrain